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ABSTRACT 


Multiple choice questions [MCQs’] form one of the tools for evaluation of medical graduates. The widespread use of MCQs’ 
raises the question of their validity and authenticity. The present study was focused on MCQ validation with an attempt to high¬ 
light the importance of item analysis and discuss the myths and realities of MCQs’. 

Aim: The aim was to perform Item analysis while probing into myths and realities of MCQ as an assessment tool for first year 
medical students. A total of hundred students underwent MCQ test as part of their first periodic assessment. 

Methodology: A total of hundred students underwent MCQ test as part of their first periodic assessment. A pre-validated MCQ 
test was given to students and a post-validation was done through item analysis. The indices were calculated using Microsoft 
excel and various indices compared. 

Results: Inspite of using a well-designed MCQ test validated by senior faculty and colleagues, after item analysis the difficulty 
index of 65 % items and discrimination index of 75 % items was in the acceptable range. The distractor efficacy showed that 9 
out of 120 distractors needed revision. 

Conclusions: Item analysis must be routinely implemented to validate MCQs’. The myths like shortfalls of MCQ framing, only 
four options in clinical setting or cueing effect are duly considered and the realities like strict and elaborate protocol for MCQ 
construction, stimulating critical and lateral thinking of students should be implemented. An integrated approach can help to 
achieve an ideal assessment tool for the benefit of students. 
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INTRODUCTION 

Evaluation is an important tool that guides student learning. 
Multiple choice questions form one of the tools for evalua¬ 
tion and are currently being widely used for assessment of 
medical graduates.‘Multiple choice questions are particu¬ 
larly useful to assess a large body of material for many stu¬ 
dents. A multiple-choice question (MCQ) consists of a stem 
with a question line at its end or underneath it, followed by 
a number of options. One of the options is correct or best 
response known as the key, while the others are described 
as distractors.‘Constructing a valid and effective MCQ is 
difficult task and needs a different approach. The methodi¬ 
cal development of MCQ banks can be an asset which can 
lead to reliable assessment of students. 2 Current widespread 


use of MCQs’ raises the issue regarding their validity and 
authenticity . 3 Review of literature highlights different views 
regarding use of MCQ or combined approach of MCQ and 
free response essay type questions for better assessment of 
medical graduates. Considering the above issues the present 
study was focused on MCQ validation through item analysis. 
The importance of item analysis and the myths and realities 
concerned with MCQ as an assessment tool for medical stu¬ 
dents are discussed further. 


MATERIAL AND METHODS 

The study was conducted in the Department of Anatomy 
as a part of First Periodic Examination. Hundred First-year 
MBBS students were voluntarily involved in the study. They 
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were given the MCQ test comprising of forty [40] questions 
with single best response. The MCQs’ were constructed to 
assess various levels of knowledge according to Bloom’s 
Taxonomy.Pre-validation of the MCQs’ was done by the 
Head of Department and other colleagues in the department. 
There was no negative marking and the time allotted was 
forty minutes. Evaluation was done out of forty marks and 
students were grouped as low, moderate and high achievers. 

Post validation of the MCQs’ was done by item analysis. 4 The 
scores of all the students were arranged in such a way that 
there was order of merit.The students were divided into 
three groups according to their marks. Those students who 
belonged to the first group were considered as high achiev¬ 
ers H [n=30] and those in the third group as low achievers 
L[n=30]. Each item was analyzed for the following; 

• Difficulty Index or Facility value or p val¬ 
ue was calculated using the formula 

p = H + L/N xlOO, 

Where H= number of students answering the item cor¬ 
rectly in the high achieving group 
L= number of students answering the item correctly in 
the low achieving group 1 

N= Total number of students in the two groups (in¬ 
cluding non-responders) 

Items having p value between 30 - 70% are consid¬ 
ered as acceptable, among which items with p value 
between 50-60% are ideal while items with p value 
less than 30% (too difficult) and more than 70% (too 
easy) are not acceptable and need modification. 1 

• Discrimination index (DI) or d value was 
calculated using the formula 

d= H-Lx 2/N 

Where the symbols H, L and N represent the same val¬ 
ues as mentioned above. 

The DI or d value is a measure of the item to discrimi¬ 
nate between students of higher and lower abilities and 
ranges between 0 and 1. In generaPd’ values between 
0.20 and 0.35 is considered as good. Items having DI 
values more than 0.35 are considered to be excellent 
and those with less than 0.20 are considered poor. 

• Distractor Effectiveness (DE) or Functionality of the 
distractors was calculated by observing how many stu¬ 
dents used the distractor in an item. If less than 5% of 
students used the option other than the key then that 
distractor is said to be a non- functional distractor 
[NFD]. On the basis of number of NFDs in an item, 
DE ranges from 0 to 100%. If an item contains three 
or two or one or nil NFDs then DE would be 0, 33.3%, 
66.6% and 100% respectively. 6 


RESULTS 

1] Difficulty Index of items analyzed 

After individually analyzing the items it was observed that 
the difficulty index (p value) of 26 items that means 65 % 
were acceptable. 11 items were too easy and 3 were too dif¬ 
ficult hence 14 itemsthat means 35 %could be used after 
modification. [Table 1] 

2] Discrimination index of items analyzed 

The discrimination index of 30 items [75 %] was acceptable 
but 10 items [25 %] need revision due to poor index. [Table 
2 ] 

3] Distractor efficacy [DE] based on number of 
NFDs’ 

As per tables 3, 4 and Graph 1, distractor efficacy was 100% 
in 24(60%) itemsand was 66.66% in 12(30%) items. So 
these questions can be added to the MCQ bank. In 3 items 
distractor efficacy was 33.33% hence their distractors should 
be modified and in 1 item having distractor efficacy 0%, its 
distractors should be either changed or the question should 
be discarded. Hence out of 120 distractors 9 needrevision. 

It was observed that higher the Difficulty Index, lower was 
the difficulty of the question. The Difficulty Index and Dis¬ 
crimination Indexwere often reciprocally related. Questions 
having a high p value (easier questions) discriminate poorly; 
while those with a low p value were good discriminators. 


DISCUSSION 

Good teaching is more a giving of right questions than a giv¬ 
ing of right answers - Joseph Alberts 

The selection of an appropriate assessment method for stu¬ 
dents’ performances remains a daunting task for the teach¬ 
ers. In an attempt to change and upgrade the existing assess¬ 
ment tools MCQs’ are introduced at various levels and are 
used rampantly for regular assessments as well as important 
career deciding entrance examinations. A large number of 
educational research studies have demonstrated poor valid¬ 
ity of MCQ’s. Barrows and Tamblyn in their book on Prob¬ 
lem Based Learning mention, multiple choice and true/false 
questions under the heading “reliable evaluation tools with 
questionable validity”. 5 Item analysis is truly the assessment 
of the assessment tool itself. 'Post examination analysis of the 
MCQs helps to assess the quality of individual test items and 
test as a whole. Poor items can be modified or removed from 
the question bank. It also helps the teachers to identity the 
subject content which lacks understanding and need greater 
emphasis and clarity, by improving or changing the method- 
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ology of teaching. In spite of validating the MCQ items by 
seniors and colleagues in the current study post validation 
systematic item analysis showed lacunae. The difficulty in¬ 
dex was acceptable in only 65 % items while the discrimi¬ 
nation index was acceptable in 75 % items. The distractor 
effectiveness also reflected need for revision in 9 out of 120 
total distractors. Similar results were seen in different stud¬ 
ies conducted for item analysis of MCQ. 6 They reveal that 
though item analysis was traditionally considered as tedious 
and time consuming task, the use of software applications 
such as Microsoft Excel has made it relatively easy and ef¬ 
fective way to calculate various indices and test the validity 
of the items. 7 Item analysis thus gives an insight and oppor¬ 
tunity to revise and improve the MCQ and develop a valid 
question bank. 1 

MCQ- Myths and Realities 

There are many opinions and views of teachers as well as 
students regarding MCQ assessment and assessment in the 
form of short essay questions. 8 This has led to the following 
discussion on the myths and realities about MCQs’. 

• MCQ as guess items assess only factual knowledge: 
One of the myth we frequently come across regarding 
MCQs’ is that they are multiple guess items 36 which 
are useful to assess only factual recall of knowledge 
and higher orders of cognitive skills cannot be as¬ 
sessed. However the reality is MCQs’ can be framed 
and well validated in order to assess higher domains of 
Bloom’s taxonomy. Developing a good bank of valid 
and reliable multiple-choicetest items targeting higher 
cognitive abilities while conformingto item construc¬ 
tion guidelines presents a big challenge to theitem 
developer. 2 Yet in reality a dilemma remains during 
assessment because MCQs’ give four options to the 
students. Hence we cannot know exactly whether a 
particular question was answered by rapid retrieval of 
overlearned content or by purposeful relating of prin¬ 
ciples. 9 

• The MCQs’ challenge - Another widely prevalent 
myth regarding use of MCQ as an assessment tool is 
that MCQs’ are easy to frame and less time consuming. 
In reality it is a challenge for the teachers to construct 
ideal MCQs’ such that all options present as plausible 
answers. The distractors should be such that they at¬ 
tract students who do not know the answer while who 
know the right answer ignore them. 

• Cueing effect - MCQs’ are recently used in most of 
the entrance examinations at various levels. This has 
probably led to the myth that they are the best and fast 
method for assessment in competitive examinations. 
Multiple studies show that if same group of students 
are assessed by MCQ and free response tests all stu¬ 
dents scored higher in objective assessment. The cue¬ 


ing effect of readymade options available was thought 
to be the major factor. 310 ’ 11 

• Integrated approach- If we consider medical profes¬ 
sional trainingit is mandatory that more emphasis be 
given to academic excellence and clinical competency. 
The medical students should be trained to evolve as 
problem solvers and not wait for situations with four 
options. Moreover there are certain clinical scenarios 
where there is one and only one diagnosis or some 
other conditions where there are more than four dif¬ 
ferential diagnoses which cannot be dealt with in an 
MCQ.The above dilemmas can perhaps be solved by 
introducing MCQs’ as a part of student assessment in 
undergraduate examinations more so in pre-clinical 
subjects. Some of the advantages in doing so are that 
the MCQs’ can be assessed rapidly with the aid of 
computers and softwares so can be employed when 
large number of students are to be assessed along with 
fast results and less error. They can provide highly 
structured and clear tasks with easy, objective and reli¬ 
able scoring where legibility of handwriting doesn’t 
matter. Such data can be compared from class to class 
and year to year. Such assessment with MCQs’ can 
also be applied to certain clinically relevant topics like 
Paediatrics. 12 Later in the medical coursewhile using 
MCQ as an assessment tool for students they should 
undergo strict protocol for their framing, validation 
and selection. 2 They should be overweighed by free 
response questions 13 and newer ways of assessment to 
enhance critical thinking of students like case based 
learning, early clinical exposure, clinical scenarios 
and problem solving can be introduced. The MCQ test 
in entrance examinations can be supplemented with 
well-designed OSCE modules to assess students’ per¬ 
formances. 14 - 15 

In studies conducted by various authors 16,17 it was seen that 
most of the students had a consistent performance in both 
objective and theory assessment. Interestingly it was also 
noticed that in some cases there was no correlation between 
MCQ and long essay scores for either the more competent 
students or the students who received failing grades. Thus, 
students’ grades detennined by an examination format that 
includes both testing modalities may be different than the 
grades obtained by using only one of the modalities. 18 


CONCLUSION 

Item analysis enhances the quality of MCQs’ which can 
help create an authentic and validated MCQ bank which 
can also assess higher orders of cognitive skills. The myths 
like shortfalls of MCQ framing, only four options in clinical 
setting or cueing effect must be considered during the use 
of MCQ as an assessment tool. The realities like strict and 
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elaborate protocol for MCQ construction, integration with 
well designed free response and short essay type questions, 
stimulating critical and lateral thinking of students should be 
implemented. Sucha perspectivecan help to achieve an ideal 
assessment tool for the benefit of students at large. 


Table 1: Difficulty Index of items analyzed 


DI Range Number of Interpretation Action tal 
items 


30 to 70 % 

26 [ 65 %] 

Acceptable 

In MCQ bank 

>70 % 

11 [ 27 . 5 %] 

Too easy 

Revise 

<30 % 

3 [7-5%] 

Too difficult 

Revise 


Table 2: Discrimination index of items analyzed 

Range Number of Interpretation 

Action taken 

items 


0.2 to 0.35 13 Good 

In MCQ bank 

> 0.35 17 Excellent 

In MCQ bank 

< 0.2 10 Poor 

Revise 


Table 3: Distractor efficacy 

Distractor type 

Number of items 

Action taken 

Functional 

99 

In MCQ bank 

Non-functional NFD 

21 

Revise 

Total 

120 



Table 4: Distractor efficacy [DE] based on number of 
NFDs’ 


NFD 

Number of items 

DE 

Action taken 

O 

24 

100 % 

In MCQ bank 

1 

12 

66 . 66 % 

In MCQ bank 

2 

3 

33 . 33 % 

Revise 

3 

1 

O 

Revise / Discard 


Distractor Efficacy based on NFDs' 

4 

V 

DE 

Blue:100% 


Red: 66% 


Green: 33.3% 

24 

Purple: 0% 


Graph 1 : Distractor efficacy [DE] and number of items 
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